Smallest Effect Size Of Interest Research Articles

The effects of oxytocin administration on social and routinized behaviors in autism: A preregistered systematic review and meta-analysis

Oxytocin administration has demonstrated considerable promise for providing individualized support for autistic people. However, studies evaluating the effects of oxytocin administration on autistic characteristics have yielded inconsistent results. This systematic review and meta-analysis investigates the effect of oxytocin administration on social and routinized behaviors in autism using recently developed methods to accurately assess potential impact of effect size dependency and publication bias. Our frequentist meta-analysis yielded a significant summary estimate for the effect of oxytocin administration on social outcomes in autism (d = 0.22, p < 0.001). The summary effect size estimate for routinized behavior outcomes was not statistically significant (d = 0.14, p = 0.22), with follow up tests indicating that the effect size estimate was not either statistically equivalent (Z = -1.06, p = 0.2), assuming a smallest effect size of interest of 0.25. Frequentist and Bayesian assessments for publication bias, as well as results from Robust Bayesian meta-analysis of oxytocin effects on social outcomes in autism, indicated that the summary effect size might be inflated due to publication bias. Future studies should aim to reduce bias by preregistering analysis plans and to increase precision with larger sample sizes.

Simulation-Based Power Analyses for the Smallest Effect Size of Interest: A Confidence-Interval Approach for Minimum-Effect and Equivalence Testing

Effect sizes are often used in psychology because they are crucial when determining the required sample size of a study and when interpreting the implications of a result. Recently, researchers have been encouraged to contextualize their effect sizes and determine what the smallest effect size is that yields theoretical or practical implications, also known as the “smallest effect size of interest” (SESOI). Having a SESOI will allow researchers to have more specific hypotheses, such as whether their findings are truly meaningful (i.e., minimum-effect testing) or whether no meaningful effect exists (i.e., equivalence testing). These types of hypotheses should be reflected in power analyses to accurately determine the required sample size. Through a confidence-interval-focused approach and simulations, I show how to conduct power analyses for minimum-effect and equivalence testing. Moreover, I show that conducting a power analysis for the SESOI might result in inconclusive results. This confidence-interval-focused simulation-based power analysis can be easily adopted to different types of research areas and designs. Last, I provide recommendations on how to conduct such simulation-based power analyses.

Equivalence of telehealth and face-to-face administration of the Wechsler Adult Intelligence Scale Fourth Edition (WAIS-IV)

Objective: Digital provision of cognitive tests like the Wechsler Adult Intelligence Scale Fourth Edition (WAIS-IV) has the potential to significantly increase access to important assessments. However, limited empirical evidence exists for the equivalence of telehealth and face-to-face administration. Presently test publishers recommend not administering subtests with stimulus materials that require manipulation via telehealth. Therefore, this study evaluated the equivalence of a telehealth administration procedure of the WAIS-IV with face-to-face administration. Method: A randomized repeated measures design included a sample of N = 28 participants with typical cognitive functioning, predominantly female (61%), aged 21 years (SD = 3.65), and identified as Australian (79%). Results: Our analysis showed that the point estimates of mean differences for indices and subtests (except PSI and Symbol Search) between face-to-face and telehealth applications were smaller than the smallest effect size of interest (SESOI). Analysis of 90% CIs around the mean difference showed the PRI Index and subtests Vocabulary, Information, and Arithmetic were statistically equivalent, while FSIQ, VCI, FRI, WMI indices, and other seven subtests were not statistically equivalent. For Null Hypothesis Significance Tests, the indices and all subtests were not significantly different from zero. Conclusions: These findings show a telehealth administration of the WAIS-IV provides scores similar to those collected in face-to-face administration, and observed differences were smaller than the difference expected due to measurement error. However, psychologists are cautioned not to solely rely on test scores when formulating outcomes but use clinical judgement with awareness of potential (albeit small) errors introduced by telehealth testing.

The Effect of Brooding About Societal Problems on Conspiracy Beliefs: A Registered Report

This Stage 2 Registered Report concerns the relationship between rumination, a repetitive style of negative thinking, and conspiracy beliefs (Stage 1 protocol: https://osf.io/y82bs, date of in-principle-acceptance: 23/05/2023). Based on four pilot studies, we tested in a fifth, registered study whether brooding, a particularly dysfunctional form of rumination, contributes to conspiracy beliefs using a repeated-measures within-person experiment (N = 1,638). Mean difference scores (conspiracy beliefs at T2 minus conspiracy beliefs at T1) were significantly greater in the brooding condition than in the control condition. However, we could neither confirm that this effect was larger than the specified smallest effect size of interest of d = 0.20, nor conclude that it was too small to be of interest (i.e., smaller than d = 0.20). We explored how reflection, an analytic form of rumination, impacted conspiracy beliefs. We further discuss implications for theories about the formation of conspiracy beliefs, and efforts aimed at preventing or reducing unfounded conspiracy beliefs. Hopefully, this article sparks a discussion among conspiracy belief researchers about how smallest effect sizes of interest could be determined in a principled way based on real-world outcomes.

Does Health Vulnerability Predict Voting for Right-Wing Populist Parties in Europe?

Abstract: Introduction: A recent article by Kavanagh et al. (2021) in American Political Science Review suggested that health vulnerability predicts voting patterns for the populist far-right. Aim: We sought to distinguish socioculturally conservative from antielite voting, expecting health vulnerability to predict only the former. Methods: We combined data from the European Social Survey (round 7) with the Chapel Hill Expert Survey (2014). We preregistered several other hypotheses regarding associations between health and voting and predetermined the smallest effect size of interest (SESOI: r = .05). Results: Health vulnerability did not predict socioculturally conservative voting, and the results for antielite voting were mixed. To investigate why our result differed from Kavanagh et al.’s, we reran their analysis employing covariate specification curve analyses. The statistical significance and the direction of the association between health vulnerability and voting depended on which covariates were included. Across 1,000 models with randomly drawn covariate specifications, 59.6% showed a positive, 29.8% a zero, and 10.6% a negative association between health vulnerability and voting for far-right populist parties. However, all effect sizes were more consistently smaller than our predetermined SESOI. Conclusions: Our study illustrates the necessity of causally justifying and preregistering all covariates and predetermining an SESOI.

Consensus meetings will outperform integrative experiments.

We expect that consensus meetings, where researchers come together to discuss their theoretical viewpoints, prioritize the factors they agree are important to study, standardize their measures, and determine a smallest effect size of interest, will prove to be a more efficient solution to the lack of coordination and integration of claims in science than integrative experiments.

Hostility has a trivial effect on persuasiveness of rebutting science denialism on social media

Polarised social media debates between science deniers and advocates for science frequently devolve into hostilities. We conducted four preregistered experiments (N = 3226; U.S. Americans) to assess how hostility influences the impact of misinformation from science deniers and rebuttals from advocates in social media discussions about vaccination (Experiment 1–3) and genetically modified food (Experiment 4). Results revealed only trivial to small effects of hostility on the persuasiveness of discussants: most internal meta-analyses revealed that the effects of hostility were smaller than the smallest effect size of interest (SESOI: d = 0.2). Thus, rebuttal is effective in mitigating the impact of misinformation on attitudes towards behaviours dismissed by deniers (for example, vaccination) and intentions to perform these behaviours, even if advocates for science lose their temper. Not responding had negative effects. Likewise, misinformation was impactful even if presented in a hostile tone. Hostility, however, consistently reduced the perceived competence of messages.

Comparing the reliability and validity of youth-reported checklists and standardized interviews for categorical measurement of emotional and behavioral problems.

Self-completed checklists measuring youth mental health problems produce dimensional scale scores and can be converted to categorical classifications representing the presence/absence of psychopathology. We test whether categorical classifications from scale scores are equivalent psychometrically to categorical classifications of the same problems obtained by lay-administered standardized structured diagnostic interviews. The sample of n = 325 youth aged 12-18 (44% male) and their parent/caregivers come from combined test-retest reliability studies conducted in Ontario, Canada, from 2011 to 2015. Ontario Child Health Study Emotional Behavioural Scales-Brief Version (OCHS-EBS-B)scores converted to categorical classifications of emotional and behavioral problems were compared with interview classifications. We test hypotheses of statistical equivalence and inferiority, using a confidence interval approach to detect if differences lie within the smallest effect size of interest of ±0.18. We compare categorical classifications on: (1) test-retest reliability (ҡ), (2) content validity (between-instrument agreement), and (3) construct validity (strength of association with three mental health-related constructs). Average test-retest reliabilities were 0.695 (checklists) and 0.670 (interviews). The reliability of checklist emotional problem classifications was not inferior to interview classifications and the difference in reliability between instruments for behavioral problems was small (-0.036). Average between-instrument agreement was ҡ = 0.586 (observed) and ҡ = 0.841 (corrected for attenuation due to measurement error) indicating high content overlap. Statistical equivalence criteria were met in 5 of 6 construct validity comparisons. Categorical classifications of emotional and behavioral problems from youth-reported checklists are, on balance, equivalent to interview classifications. Checklists represent a simple, brief, inexpensive alternative to interviews.

No evidence that Chinese playtime mandates reduced heavy gaming in one segment of the video games industry.

Governments around the world are considering regulatory measures to reduce young people's time spent on digital devices, particularly video games. This raises the question of whether proposed regulatory measures would be effective. Since the early 2000s, the Chinese government has been enacting regulations to directly restrict young people's playtime. In November 2019, it limited players aged under 18 to 1.5 hours of daily playtime and 3 hours on public holidays. Using telemetry data on over seven billion hours of playtime provided by a stakeholder from the video games industry, we found no credible evidence for overall reduction in the prevalence of heavy playtime following the implementation of regulations: individual accounts became 1.14 times more likely to play heavily in any given week (95% confidence interval 1.139-1.141). This falls below our preregistered smallest effect size of interest (2.0) and thus is not interpreted as a practically meaningful increase. Results remain robust across a variety of sensitivity analyses, including an analysis of more recent (2021) adjustments to playtime regulation. This casts doubt on the effectiveness of such state-controlled playtime mandates.

Protocol Standardization May Improve Precision Error of InBody 720 Body Composition Analysis.

Bioelectrical impedance analysis (BIA) is a popular technique which can be used to track longitudinal changes in body composition. However, precision of the technique has been questioned, especially among athletic populations where small but meaningful changes are often observed. Guidelines exist which attempt to optimize precision of the technique but fail to account for potentially important variables. Standardization of dietary intake and physical activity in the 24hr prior to assessment has been proposed as an approach to minimizing the error of impedance-derived estimates of body composition. Eighteen recreational athletes, male (n = 10) and female (n = 8), underwent two consecutive BIA tests to quantify within-day error, and a third test (the day before or after) to quantify between-day error. All food and fluid intake plus physical activity from the 24hr prior to the first BIA scan was replicated during the following 24hr. Precision error was calculated as the root mean square standard deviation, percentage coefficient of variation, and least significant change. There were no significant differences in precision error of within- and between-day fat-free mass, fat mass, and total body water. Differences in precision error of fat-free mass and total body water, but not fat mass, were less than the smallest effect size of interest. The 24-hr standardization of dietary intake and physical activity may be an effective approach to minimizing precision error associated with BIA. However, further research to confirm the validity of this protocol compared to nonstandardized or randomized intake is warranted.

An evolutionary model for aggression in youth: Rethinking aggression in terms of the Catalyst Model

Like many areas of psychology, aggression research has been impacted by psychology's replication crisis. Until recently, aggression was understood largely from the perspective of social cognitive models, wherein aggression was conceptualized as learned scripts. However, more recent rigorous research has suggested that cognitive scripts may play only a small role in aggression. By contrast, genetic and evolutionary inputs are powerful and combine with both developmental and situational life stress to increase the risk for serious aggression. This article examines aggression from the Catalyst Model and how diathesis-stress approaches to understanding aggression can help reduce error in predicting clinically significant aggression. This article also addresses related issues of measurement error and smallest effect size of interest (SESOI) as they relate to aggression research.

Low Prevalence of A Priori Power Analyses in Motor Behavior Research

A priori power analyses can ensure studies are unlikely to miss interesting effects. Recent metascience has suggested that kinesiology research may be underpowered and selectively reported. Here, we examined whether power analyses are being used to ensure informative studies in motor behavior. We reviewed every article published in three motor behavior journals between January 2019 and June 2021. Power analyses were reported in 13% of studies (k = 636) that tested a hypothesis. No study targeted the smallest effect size of interest. Most studies with a power analysis relied on estimates from previous experiments, pilot studies, or benchmarks to determine the effect size of interest. Studies without a power analysis reported support for their main hypothesis 85% of the time, while studies with a power analysis found support 76% of the time. The median sample sizes were n = 17.5 without a power analysis and n = 16 with a power analysis, suggesting the typical study design was underpowered for all but the largest plausible effect size. At present, power analyses are not being used to optimize the informativeness of motor behavior research. Adoption of this widely recommended practice may greatly enhance the credibility of the motor behavior literature.

Impact of 24-Hr Diet and Physical Activity Control on Short-Term Precision Error of Dual-Energy X-Ray Absorptiometry Physique Assessment.

Dual-energy X-ray absorptiometry (DXA) is a popular technique used to quantify physique in athletic populations. Due to biological variation, DXA precision error (PE) may be higher than desired. Adherence to standardized presentation for testing has shown improvement in consecutive-day PE. However, the impact of short-term diet and physical activity standardization prior to testing has not been explored. This warrants investigation, given the process may reduce variance in total body water and muscle solute, both of which can have high daily flux amongst athletes. Twenty (n = 10 males, n = 10 females) recreationally active individuals (age: 30.7 ± 7.5years; stature: 176.4 ± 9.1cm; mass: 74.6 ± 14.3kg) underwent three DXA scans; two consecutive scans on 1day, and a third either the day before or after. In addition to adhering to standardized presentation for testing, subjects recorded all food/fluid intake plus activity undertaken in the 24hr prior to the first DXA scan and replicated this the following 24hr. International Society of Clinical Densitometry recommended techniques were used to calculate same- and consecutive-day PE. There was no significant difference in PE of whole-body fat mass (479g vs. 626g) and lean mass (634g vs. 734g) between same- and consecutive-day assessments. Same- and consecutive-day PE of whole-body fat mass and lean mass were less than the smallest effect size of interest. Inclusion of 24-hr standardization of diet and physical activity has the potential to reduce biological error further, but this needs to be verified with follow-up investigation.

COVID anxiety and its predictors among Slovak adolescents.

The COVID-19 pandemic and its related restrictions, mainly social distancing, had an impact on the mental health of various groups, including adolescents. The main goal of our study was to explore the impact of gender, age, resilience (measured using the Brief Resilience Scale), attachment anxiety, attachment avoidance (both measured using the Experiences in Close Relationships Revised Scale for adolescents), and mental and general health (measured using items of SF-8 Health Survey) on COVID anxiety (measured using the COVID Anxiety Scale) among a sample of Slovak adolescents (N = 1,786, age 15 to 19, mean age = 16.8, SD = 1.2). The data were collected online between 13 April and 24 May 2021. Four nested linear regression models were fitted to the data and evaluated. The significant predictors that had a greater effect than our smallest effect size of interest (β  = 0.10) were gender (β  = -0.26, p  < 0.001, where boys had lower scores in COVID anxiety), general and mental health (β  = -0.13 and β  = -0.14, respectively, both with p  < 0.001), resilience (β  = -0.12, p  < 0.001), and attachment avoidance (β  = -0.11, p  < 0.001). Similarly, age and attachment anxiety were significant predictors with a lower effect size (β  = 0.06, p  = 0.003, and β  = 0.09, p  < 0.001, respectively). Our results are in line with previous research findings highlighting the importance of prevention and interventions programs focused mainly on preventing loneliness and social disconnection, fostering secure attachment with parents and peers, and increasing the resilience of adolescents, especially in the stressful time of a pandemic, to promote their mental health.

The importance of the smallest effect size of interest in expert witness testimony on alcohol and memory.

Memory experts are sometimes asked to evaluate the validity of accounts of witnesses, victims, or suspects. In some of these cases, they are asked what effect alcohol has on the validity of such accounts. In this article, we offer a guide on what expert witnesses can reliably say about how alcohol affects memory. We do so by resorting to effect sizes from previous studies and meta-analytic work, and address this novel question: Are these effect sizes meaningful in legal cases? More specifically, we argue that any determination of whether individual studies about alcohol and memory are practically relevant for legal cases, scientists must focus on the smallest effect size of interest. We make the case that a decrease or increase of only 1 detail, especially an incorrect detail, should be regarded as the smallest effect size of interest in this line of research. In line with this idea, we show that effect sizes in the alcohol and memory literature are often larger than this smallest effect size of interest. This finding is important because it implies that alcohol often exerts a practically relevant and meaningful detrimental effect on the reporting of both correct and incorrect details, which in turn negatively affects the validity of witness testimony.

Power to the People: A Beginner’s Tutorial to Power Analysis using jamovi

Authors have highlighted for decades that sample size justification through power analysis is the exception rather than the rule. Even when authors do report a power analysis, there is often no justification for the smallest effect size of interest, or they do not provide enough information for the analysis to be reproducible. We argue one potential reason for these omissions is the lack of a truly accessible introduction to the key concepts and decisions behind power analysis. In this tutorial targeted at complete beginners, we demonstrate a priori and sensitivity power analysis using jamovi for two independent samples and two dependent samples. Respectively, these power analyses allow you to ask the questions: “How many participants do I need to detect a given effect size?”, and “What effect sizes can I detect with a given sample size?”. We emphasise how power analysis is most effective as a reflective process during the planning phase of research to balance your inferential goals with your resources. By the end of the tutorial, you will be able to understand the fundamental concepts behind power analysis and extend them to more advanced statistical models.

The Effects of Training Load During Dietary Intervention Upon Fat Loss: A Randomized Crossover Trial

ABSTRACTPurpose: To date no studies have compared resistance training loading strategies combined with dietary intervention for fat loss. Methods: Thus, we performed a randomised crossover design comparing four weeks of heavier- (HL; ~80% 1RM) and lighter-load (LL; ~60% 1RM) resistance training, combined with calorie restriction and dietary guidance, including resistance trained participants (n=130; males=49, females=81). Both conditions performed low-volume, (single set of 9 exercises, 2x/week) effort matched (to momentary failure), but non-work-matched protocols. Testing was completed pre- and post-each intervention. Fat mass (kg) was the primary outcome, and a smallest effect size of interest (SESOI) was established at 3.3% loss of baseline bodyweight. Body fat percentage, lean mass, and strength (7-10RM) for chest press, leg press, and pull-down exercises were also measured. An 8-week washout period of traditional training with normal calorie interspersed each intervention. Results: Both interventions showed small statistically equivalent (within the SESOI) reductions in fat mass (HL: -0.67 kg [95%CI -0.91 to 0.42]; LL: -0.55 kg [95%CI -0.80 to -0.31]) which were also equivalent between conditions (HL – LL: -0.113 kg [95%CI -0.437 kg to 0.212 kg]). Changes in body fat percentage and lean mass were also minimal. Strength increases were small, similar between conditions, and within a previously determined SESOI for the population included (10.1%). Conclusions: Fat loss reductions are not impacted by resistance training load; both HL and LL produce similar, yet small, changes to body composition over a 4-week intervention. However, the maintenance of both lean mass and strength highlights the value of resistance training during dietary intervention.

Volleyball Competition on Consecutive Days Modifies Jump Kinetics but Not Height.

In volleyball, jump execution is critical for the match outcome. Game-play-related neuromuscular impairments may manifest as decreased jump height (JH) or increased jump total duration, both of which are pivotal for performance. To investigate changes in JH and kinetics with game play, the authors conducted a prospective exploratory analysis using minimal-effect testing (MET) and equivalence testing with the 2 one-sided tests procedure, univariate, and bivariate functional principal component analysis, respectively. Twelve male varsity athletes completed 3-set matches on 2 consecutive days. Countermovement jumps were performed on a force platform immediately prematch and postmatch on days 1 and 2 and once on days 3 and 4. Across sessions, JH was equivalent (P < .022, equivalence test), while total duration reported inconclusive changes (P > .227). After match 2, MET indicated that relative force at zero velocity (P = .036) decreased, while braking duration (P = .040) and time to peak force (P = .048) increased compared with baseline. With the first and second functional principal components, these alterations, together with decreased relative braking rate of force development (P = .092), were already evident after match 1. On day 4, MET indicated that relative peak force (P = .049), relative force at zero velocity (P = .023), and relative braking rate of force development (P = .021) decreased, whereas braking duration (P = .025) increased from baseline. Impairments in jump kinetics were evident from variables related to the countermovement-jump braking phase, while JH was equivalent. In addition to these experimental findings, the present research provides information for the choice of sample size and smallest effect size of interest when using MET and 1- and 2-dimensional analyses for countermovement-jump height and kinetics.

Sample Size Justification

An important step when designing an empirical study is to justify the sample size that will be collected. The key aim of a sample size justification for such studies is to explain how the collected data is expected to provide valuable information given the inferential goals of the researcher. In this overview article six approaches are discussed to justify the sample size in a quantitative empirical study: 1) collecting data from (almost) the entire population, 2) choosing a sample size based on resource constraints, 3) performing an a-priori power analysis, 4) planning for a desired accuracy, 5) using heuristics, or 6) explicitly acknowledging the absence of a justification. An important question to consider when justifying sample sizes is which effect sizes are deemed interesting, and the extent to which the data that is collected informs inferences about these effect sizes. Depending on the sample size justification chosen, researchers could consider 1) what the smallest effect size of interest is, 2) which minimal effect size will be statistically significant, 3) which effect sizes they expect (and what they base these expectations on), 4) which effect sizes would be rejected based on a confidence interval around the effect size, 5) which ranges of effects a study has sufficient power to detect based on a sensitivity power analysis, and 6) which effect sizes are expected in a specific research area. Researchers can use the guidelines presented in this article, for example by using the interactive form in the accompanying online Shiny app, to improve their sample size justification, and hopefully, align the informational value of a study with their inferential goals.

Within- and between-person effects of naltrexone on the subjective response to alcohol and craving: A daily diary investigation.

Naltrexone is an effective treatment for heavy drinking among young adults. Laboratory-based studies have shown that naltrexone dampens the subjective response to alcohol and craving. However, few studies have tested naltrexone's dynamic, within-person effects on subjective response and craving among young adults in natural drinking environments. Using daily diary data from a randomized, placebo-controlled study of naltrexone's efficacy in young adults, we examined the between-person effects of treatment condition (i.e., naltrexone vs. placebo) and medication dosage (i.e., daily, targeted, and daily+targeted) on the subjective response to alcohol and craving on drinking days. Multilevel mediation models predicted subjective response and craving from treatment condition (between-person) and medication dosage (within-person), accounting for drinking levels. All effects were disaggregated within and between persons. At the between-person level, naltrexone directly blunted intense subjective effects (i.e., "impaired", "drunk") and indirectly blunted subjective effects through reduced drinking. Naltrexone was not associated with craving. Between-person effects were not significant after alpha correction, but their effect sizes (bs=0.14 to 0.17) exceeded the smallest effect size of interest. At the within-person level, taking two (vs. 1) pills was associated with heavier drinking, and taking one (vs. 0) pill was associated with lighter drinking, and lighter drinking was associated with a lower subjective response and craving. Treatment condition did not moderate the within-person effects of dosing on outcomes. Our findings suggest that the direct between-person effect of naltrexone was largest on intense subjective responses, blunting feelings of being "drunk" and "impaired". Future research using momentary (rather than daily) assessments could confirm and extend these findings.